A canonical quality assurance procedure typically tests common code paths, which are paths most often executed by the software system as it responds to routine operations. The testing of common code paths may not provide adequate testing coverage. For example, the exception and failure of modules, uncommon interleaving of threads, and missing optional fields may not be properly tested when testing common code paths. For the exception example, software programs may use exceptions, but many throw clauses for the exceptions are not exercised by routine testing. For example, an exception may not be properly caught or handled by upstream modules. For a failure of a module, such as a hang or crash, the testing of the robustness and recovery mechanism of the modules that depend on the failed module may not be exercised under normal load and is difficult to troubleshoot in distributed systems.
Also, situations due to the unanticipated interleaving of process threads are very time-consuming to triage and often occur only during stress testing where the level of activity can impede figuring out the root cause. Altering the interleaving execution sequence of multiple threads may be used to discover concurrency situations including severe issues like deadlock and data corruption. However, repetitive test runs usually share the same interleaving pattern due to the similarity of test setup and environment. Typical ways of changing the interleaving execution include adding a delay in one thread when it reaches a point in the code and blocking execution until certain conditions are met. Users typically need to perform the above with debuggers, which is very time consuming due to the lack of automation support and cannot be precisely done.
For the optional fields example, public data structures have optional fields that are set in canonical test cases. Absence of optional fields in application programmer interface (API) calls might trigger different code paths and uncover situations of making invalid assumptions that the field is present. Although seemingly tested, situations resulting from improper handling of optional fields in data structure objects are often encountered.
Improving the test coverage for these abnormal execution scenarios may be challenging for a number of reasons. First, test cases are difficult to envision and design because it is not intuitive what specific scenarios among others might expose vulnerable behavior and hence should be tested. Thus, companies tend to wait for a bug to be filed before investigating an area. Unfortunately, these issues are found late in a release cycle, which adds risk to the release. Also, it is challenging to fully automate a test for any abnormal scenarios that are understood because it is hard to reliably generate the triggering event under all conditions in which the test needs to be run. Failures can be injected into program execution, but this requires mechanisms to inject failures at the right place and the relevant workflows to be built, which is not typically performed due to the pressures of meeting release dates. Additionally, these tests are not easily integrated with the testing framework that is typically used so that testing can be performed systematically. Finally, there is no measurement scheme for testing abnormal execution scenarios. Test cases are added to a testing suite manually in an ad hoc manner usually without the notion of code coverage. It is a problem to systematically conduct testing of abnormal execution scenarios with a measurable coverage criteria.
The robustness and reliability of a software system is typically verified by instrumentation of the software system so that a change of state (e.g., a failure or delay) is injected at specific points during the execution. The instrumentation involves modification of source code and requires a rebuild of the software system.